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Objectives: 


1. Norming the MindPlay Universal Screener using ELA Standards in Grades 3-9 
2. Assessing the Impact of MVRC Exposure on ELA Outcomes in Grades 3-9 
3. Exploring Factors that Modulate the Extent to which Students Benefit from MVRC 


Abstract: 


A data set from an urban Midwestern school district was mined to explore how the technology- 
based reading enrichment known as Mindplay Virtual Reading Coach (MVRC) affects children’s 
performance on the English Language Arts (ELA) Standards state-wide assessment (N = 6098 
students from Grades 3 to 9). ELA data from two times points were available, approximately one 
year apart. ELA data were correlated with various data points obtained from MVRC, including 
the benchmark assessment administered at the beginning and at the end of the year. Results 
revealed large correlations across grade levels for the MVRC Composite score obtained from the 
MVRC Universal Screener, 0.40 < r < 0.74. Results also revealed that the amount of MVRC 
exposure was linearly related to an increase in ELA performance at the end of the year, largely 
independent of grade level and students’ initial reading competence. Girls and boys benefited 
equally from MVRC exposure, as did children from different ethnicities. The most prevalent 
factor in predicting the ELA-MVRC relation was the type of school, with MVRC exposure 
having the highest benefits in non-failing elementary schools, compared to high schools. 


Highlights: 


- The outcome of the MVRC assessment battery closely tracked the outcome of the ELA 
state-wide standardized test, despite notable differences in the emphasis of the two types 
of assessments. 

- Students benefited from MVRC exposure, independent of their grade level, initial reading 
competence, gender, or identified ethnicity. 

- Schools differed considerably in the degree to which their students benefited from 
MVRC exposure (across grade level), hinting at structural barriers to learning. 


Introduction: 


MindPlay Virtual Reading Coach is a commercially available educational software 
geared towards improving reading fluency in an individualized learning environment. Lessons 
are provided by virtual reading specialists and speech pathologists, followed by online practice 
that includes immediate and specific feedback. Depending on the needs of the student, emphasis 
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is placed on phonological awareness, phonics skills, vocabulary, grammar, silent reading fluency 
and comprehension. An underlying flow-chart structure defines the order in which lessons and 
practice activities are presented. Lessons are continuously adapted to fit the needs and emerging 
skills of individual students. 

Several empirical studies have demonstrated the positive effect of MVRC exposure on 
reading skills (e.g., Bauer-Kealey & Mather, 2018; Chambers, Mather, & Stoll, 2013; Kloos, 
Sliemers, Cartwright, Mano, & Stage, 2019; Schneider et al., 2016; Vaughan, Crews, Sisk, & 
Garcia, 2004). For example, 2"'-graders who logged in for an average of 44 MVRC hours 
improved in reading fluency more than students who did not take part in the intervention 
(Schneider et al., 2016). In addition, students who were exposed to MVRC for 9 weeks in 2"! and 
4'" srade improved in reading fluency more so than students who used an alternative reading 
technology (Kloos et al., 2019). The current report was designed to further substantiate these 
findings, looking specifically at the link between MVRC and performance on the state-mandated 
ELA assessment. 


Approach: 


As part of a larger partnership, data were analyzed from an urban school district that 
serves a large group of students from economically disadvantaged communities. Table 1 
provides demographic details of the available data set. For ease of description, we have 
categorized students in terms of the number of hours they were exposed to MVRC during the 
year of the study (in increments of 10 hours). 


Table 1: Number of Students, Separated by Demographics 


MVRC Exposure (in Hours) 
<10 10-20 20-30 30-40 40-50 50-60 60+ 


Grade Level 
Grade 3 42 76 109 201 209 203 190 
Grade 4 30 68 94 111 147 145 316 
Grade 5 36 70 141 181 215 149 90 
Grade 6 100 64 114 169 184 119 91 
Grade 7 234 132 136 148 108 47 19 
Grade 8 216 141 139 93 68 39 47 
Grade 9 576 103 129 37 9 4 2 
Gender 
Female 662 277 385 468 492 379 389 
Male 571 377 477 472 450 329 367 
Race/Ethnicity 
African American 810 473 612 617 578 391 431 
White 298 106 159 225 256 228 244 
Other 126 75 91 98 108 89 81 
Special Ed Status 
No 990 501 680 768 811 611 659 
Yes 214 150 180 168 125 94 96 
N/A 30 3 2 4 6 3 1 


Objective 1: Norming the MVRC Universal Screener using ELA Standards in Grades 3-9 


The MVRC contained a comprehensive diagnostic tool, known as the Universal Screener 
(MindPlay Universal Screener!™ Resource Guide, 2018). For Grades 2 and above, it consisted of 
an assessment of reading fluency (which returns a Composite score of grade-equivalent reading 
fluency), a visual-scanning test, a listening-vocabulary test, a phonics test, and a letter- 
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discrimination test. Students also completed the mandatory ELA state assessment at two time 
points: in the Spring prior to the use of MVRC, and in the Spring after MVRC exposure. Figure 
1 shows the findings, expressed as correlation coefficients. Note that 7 =.30 represents moderate 
a correlation, while r = .50 represents a strong correlation. 


Time 1: Spring Prior to MVRC Exposure Time 2: Spring After MVRC Exposure 
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Figure 1. Correlations were carried out for each grade level separately. 
Number of students is provided in parenthesis. 


Findings. For most grade levels, the composite measure of the MVRC Universal Screener 
correlated strongly with the ELA measure. These findings were impressive, given that both 
MVRC and ELA return categorical outcomes (MVRC Composite score: grade equivalence 
score; ELA: 5-item scale; 1 = limited; 2 = basic; 3 = proficient; 4 = accelerated; 5 = advanced). It 
is also important to note that MVRC reading fluency is correlated with ELA more strongly than 
the MVRC measures of phonics, listening vocabulary, and letter discrimination, ps < .01. The 
differences between Time | and Time 2 correlations are likely due to the difference in the 
number of participants (ELA data at Time | were only available for 4043 students). 


Conclusion. MVRC’s composite score correlated highly with a state-wide assessment of 
reading skills. This speaks to the construct validity of the MVRC diagnostic tool. 


Objective 2: Assessing the Impact of MVRC Exposure on ELA Standards in Grades 3-9 


To what extent did students’ exposure to MVRC during the school year impact end-of-the- 
year ELA performance? To answer this question, students with access to MVRC were 
categorized based on the amount of time they spent on MVRC lessons and practice. Figure 2 
provides the results. 
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Figure 2. Average ELA performance at the end of the year, 
after exposure to MVRC for different lengths of time. 


Findings. There was a linear trend between the amount of MVRC exposure and 
performance on the ELA assessment at the end of the year. This trend is best visible when scores 
are added across grade levels (Grand Total). Exceptions to this trend are seen when students used 
MVRC for less than 20 hours (shown in dark and light red in Figure 2): For most grade levels, 
students with less than 10 hours exposure to MVRC performed as good or better than students 
with 10 to 20 hours of MVRC exposure. Surprisingly, the linear trend between MVRC exposure 
and ELA outcome did not hold up for 7 graders: Students who completed more than 60 MVRC 
hours nevertheless performed as poorly on the ELA assessment as students who completed very 
few MVRC hours. This is likely due to the fact that only 3% of the 7" graders completed 60 or 
more MVRC hours. 


Conclusion. In the aggregate, the time spent on MVRC activities had a positive effect on 
the outcome of the high-stake state assessment. Given that MVRC is geared towards improving 
reading fluency, this finding highlights the importance of reading fluency in achieving general 
reading proficiency. 


Objective 3: Exploring Factors that Modulate the Extent to which Students Benefit from MVRC 


Despite promising results of MVRC exposure, students differed in the extent to which 
they benefitted from MVRC. For example, in the current data set, 206 students were critically 
behind in reading proficiency at the beginning of the year but did not make any progress by the 
end of the year, despite completing over 60 MVRC hours. While this is only a small proportion 
of students, it raises the question about factors outside of MVRC exposure that affect learning. 
One way to address this question is to calculate the size of the slope in an MVRC-ELA 
regression model (i.e., predictor variable = amount of MVRC exposure; predicted variable = 
ELA outcome). A small slope indicates little predictive value, while a large slope indicates high 
predictive value. 


Findings. A comparison of slopes predicting ELA proficiency from MVRC exposure 
revealed no effect of gender (Bremale = .16; Bmale = .18): Both girls and boys benefited from 
MVRC exposure to a similar extent. Similarly, ethnicity did not affect the size of the slope 
(Belack/African American = 0.16; Bwhite/Caucasian = -22; Bother = .17). However, there was a slope 
difference between typically developing students (B = .19) and students eligible for special 
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education services (B = .11). There was also a slope difference between grades: the 7th- 
grade slope was lower than the slopes of the other grades (B3 = .20; Ba = .17; Bs= .24; Bo = 
.18; B7 = .08; Bg = .14; Bo = .17). Perhaps most striking was the difference among schools. While 
students in non-failing elementary schools benefited the most from MVRC exposure (B = .19), 
students in failing elementary schools benefited to a lesser extent (B = .14), and high school 
students benefited even less (B = .06). 


Conclusion. MVRC findings were robust across various demographics, including 
students’ gender and ethnicity. Students’ grade level was largely inconsequential as well: Students 
benefited from MVRC exposure in similar ways, whether in early or later grades. The largest 
modulating factor was the type of school: Students attending non-failing elementary schools 
benefited almost twice as much from MVRC exposure as students attending high schools. Further 
work is needed to determine how to address the structural barriers to MVRC learning. 


For questions or comments, or to obtain more detailed information about the analyses presented here, 
please send an email to heidi.kloos @uc.edu. 
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